There is a natural tendency to think that the more features that you have to describe the observations in your data, the more information you have about those observations. Thus, the more accurate your models should be. However, that is true only to a point because having more columns is not always helpful.

When the number of potential input variables is large, some of these inputs can be redundant. Redundant inputs complicate model construction. One consequence of input redundancy might be model instability.

The scatter plot between Input1 (plotted on horizontal X-axis) and Input2 (plotted on vertical Y-axis) shows that the cases with higher values on Input1 are located further to the right, and cases with higher values on Input2 are located higher up on the graph. In this scatter plot, data points are tightly clustered together, giving an impression of an upward trending straight line, meaning that as the value of Input1 increases so does a case's Input2 value.

Here Input1 and Input2 are correlated and thus redundant. Knowing the value of Input1 gives you a good idea of the value of Input2. Generally, there would not be any gain in having both in the model. Redundancy is discussed in a previous lesson.

Irrelevancy creeps in when you have many input variables. An irrelevant input does not provide information about the target and therefore does not contribute to the model. Adding irrelevant features unnecessarily complicates and increases the expense of training models.

A three-dimensional graph with Expected value of the (Target) plotted on vertical Y-axis. The two inputs, Input 2 and Input3, are plotted on X-axis and Z-axis. In this graph the Expected value of the (target) seems to be changing with the values of Input2, but hardly with the values of Input3.

Here, predictions change with Input2 but much less with Input3, and thus, Input3 is irrelevant. There would not be any gain in adding Input3 into the model. Irrelevancy is discussed in a previous lesson.

The dimension refers to the number of input variables (actually input degrees of freedom). We generally encounter large numbers of input variables in machine learning. The number of variables often has a greater effect on computational performance than the number of cases. Curse of dimensionality means that the number of dimensions is staggeringly high, so high that calculations become extremely difficult, and the number of features can exceed the number of observations.

A one-dimensional graph depicting a single input variable, color of the sheep in this case. The color of the sheep is represented on the horizontal axis where sheep are arranged in the order of shade, from lighter sheep on the extreme left to darker on the right. The sheep look closer to each other in this one-dimensional representation.

Consider a simple example of finding darker sheep. The only available feature is color of sheep. It's pretty easy for you to select perhaps the three most gray sheep. Here, data density is very high. Hence, you will not face any difficulty to trace these sheep.

In this graph you have added another dimension, size of the sheep, on the Y-axis making, it a two-dimensional graph. Now the sheep are plotted based on their color and size. The horizontal X-axis represents the color values from lighter shade to darker as you move from left to right. And the vertical Y-axis represents the size of the sheep, ranging from smaller at the bottom to larger at the top. The sheep look farther from each other in this two-dimensional representation.

If you add another feature such as size of sheep to its color, then the complexity of a data set increases. If you need to find the darker and bigger sheep, you will take a little more time as compared to previous scenario. Perhaps you select two sheep in this case. Thus, high dimensionality limits the ability to explore and model the relationships among the variables.

A three-dimensional plot where three input variables are graphed: color, size and wool density of the sheep. The X-axis represents the color values, from lighter shade on the left to darker shade on the right. The Y-axis represents the size of the sheep, from smaller sheep at the bottom to bigger sheep at the top. The Z-axis represents the wool density. The sheep look farther from each other in this three-dimensional representation.

If you add one more feature, wool density to its size and color, the complexity of a data set increases rapidly with increasing dimensionality (Breiman et al. 1984) . If you need to find the darker, bigger sheep with high wool density, you will take a little more time as compared to previous scenario. Perhaps you want to select one sheep in this case. This added dimensionality further limits your ability to explore and model the relationships among the variables.

Sparseness addresses the so-called curse of dimensionality, which tells us that as you add features, you are introducing greater sparsity to your observations in that feature space. Sparseness refers to a matrix of numbers that includes many zeros or values that will not significantly impact a calculation.

Three graphs: 1-D, 2-D, and 3-D. The one-dimensional graph, which depicts the color of eight sheep on a real number line, is densely covered with data points - that is, the data related to the colors of very high percentage sheep (8 out of 10) are available. The two-dimensional graph, which represents color on the X-axis and size of sheep on Y-axis, lacks data values for the majority of the color-size combinations. It has only 8 out of 100 observations values filled. The problem of sparsity becomes more evident as you move from 2-D to 3-D. The three-dimensional graph represents color on the X-axis, size on the Y-axis, and wool density on Z-axis. A high majority of the color-size-wool density combinations lack data values, hence the graph has a very low percentage of available data positions filled - that is, only 8 out of 1000 observations.

In this simple example, where in one dimension you have the space well covered with eight data points that covers 80% of the space, when we add another dimension, those same eight data points cover a much smaller percentage of the feature space, only 8%. This sparsity grows exponentially with each feature added. The practical meaning of sparseness is that there is much space between the data points in the high-dimensional space built up by all the used input variables. The data points have a long way to their neighbors -- that is, they are lonely. Many variables consist of one or two values (for example, 0 or 1). This is a source of problems for many machine learning methods as such matrices are computationally expensive.

Large sparse matrices are common in applied machine learning, such as in data that contains counts, data encodings that map categories to counts, and even in natural language processing. The solution to representing and working with sparse matrices is to use an alternate data structure to represent the sparse data.

The white circles on the plot represent the observed data points to which models are fit. An insufficiently complex model, shown in blue color, clearly has a smooth curve, however, it's missing the patterns (signals) present in data. Such models can lead to underfitting. On the other hand, an overly complex model, shown in red color, clearly has a jagged, less smooth curve, which can lead to overfitting. The overly complex model seems to be fitting data too well, such that it can even accommodate the randomness in the data.

Overfitting occurs when the model fits the training data too well but cannot generalize to unseen data sets. Overfitting was discussed in the previous lesson. Overfitting will be discussed in a broader sense in the next section.

A critical part of the success of a machine learning project is producing a good set of features to train on. You need to include features to characterize the observations, but it is important to be selective about the features you choose to incorporate in your modeling process. For example, with the curse of dimensionality, the amount of data that you would need to collect to get a meaningful predictive model would be enormous. Therefore, methods are needed to transform the data from this high-dimensional space to fewer dimensions. Feature engineering is the process of determining the best set of inputs to use in building predictive models.

In the context of the entire analytics life cycle, feature engineering is typically done as part of the data preprocessing leading into modeling. It is a big part of the 80% project time that you hear about being spent in preparation for modeling. And it is a very important step that can make a big difference in the accuracy and effectiveness of your models.

You need great features that describe the structures inherent in your data. The reduced dimensionality of the data set can result in simple, more accurate models that train in less time.